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1. INTRODUCTION 

The 45th Weather Squadron (45 WS) Launch 
Weather Officers (LWO's) use the 12-km 
resolution North American Mesoscale (NAM) 
model (MesoNAM) text and graphical product 
forecasts extensively to support launch weather 
operations. However, the actual performance of 
the model at Kennedy Space Center (KSC) and 
Cape Canaveral Air Force Station (CCAFS) has 
not been measured objectively. In order to have 
tangible evidence of model performance, the 45 
WS tasked the Applied Meteorology Unit (AMU; 
Bauman et al, 2004) to conduct a detailed 
statistical analysis of model output compared to 
observed values. The model products are 
provided to the 45 WS by ACTA, Inc. and include 
hourly forecasts from 0 to 84 hours based on 
model initialization times of 00, 06, 12 and 
18 UTC. The objective analysis compared the 
MesoNAM forecast winds, temperature (T) and 
dew point (T d ), as well as the changes in these 
parameters over time, to the observed values from 
the sensors in the KSC/CCAFS wind tower 
network shown in Table 1. These objective 
statistics give the forecasters knowledge of the 
model's strengths and weaknesses, which will 
result in improved forecasts for operations. 

2. BACKGROUND 

The 45 WS requested the data sets be 
stratified by year, warm season (May-September), 
cool season (October-April), month and model 
initialization time. They also requested verification 
the model forecasts for the current operational 
version of the MesoNAM. This paper will address 
the following statistics requested by the 45 WS: 

• Bias (mean difference), 

• Standard deviation of Bias, and 

• Root Mean Square Error (RMSE). 

3. WIND TOWER DATA 

The current version of the operational 
MesoNAM became available in August 2006. 


Table 1. Towers, launch activities and sensor heights at 
KSC and CCAFS used in the objective analysis to verify 
the MesoNAM forecasts. 

Tower 

Number 

Supported Activity 
and Facility 

Sensor 

Heights 

0002 

Delta II (LC-17) 

6 ft, 54 ft, 90 
ft 

0006 

Delta IV (LC-37) 

54 ft 

0108 

Delta IV (LC-40) 

54 ft 

0110 

Atlas V/Falcon (LC-41) 

54 ft, 162 ft, 
204 ft 

0041 

Atlas V (LC-41) 

230 ft 

393/394 

Shuttle/Constellation 

(LC-39A) 

60 ft 

397/398 

Shuttle/Constellation 

(LC-39B) 

60 ft 

511/512/513 

Shuttle Landing Facility 

6 ft, 30 ft 


Therefore, the period of record (POR) for this data 
set starts with the first cool season month of 
October 2006. The KSC/CCAFS wind tower data 
were acquired for the period October 2006 to April 
2009 from the AMU archive, and the AMU wind 
tower quality control (QC) software was used to 
remove erroneous observations from the dataset. 
The statistical analysis software S-PLUS® 
(Insightful Corporation 2007) was used to process 
the wind tower data. Scripts were written in S- 
PLUS to import and modify the QC’d wind tower 
observation files to remove unneeded time periods 
and sensor heights from the dataset for each 
tower. The locations of the towers used for the 
verification are shown on the map of KSC/CCAFS 
in Figure 1 . 

Since the tower data were reported every 
5 minutes and the MesoNAM forecasts were 
hourly, the 45 WS requested the AMU calculate 
the mean value for each observed parameter from 
the tower data at the top of every hour using the 
observations from 30 minutes prior and 30 
minutes after the hour. The S-PLUS scripts were 
written to reformat the tower data and calculate 
the mean values in this manner. 
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Figure 1 . Map of KSC/CCAFS showing the locations of the wind towers used to verify MesoNAM forecasts 
(red pentagons labeled with tower number and the supported launch activity), the locations of the MesoNAM 
model grid points (green circles with black dot) and CCAFS weather station (magenta square labeled KXMR). 


4. MESONAM FORECAST PRODUCTS 

The AMU requested and obtained the 
archived MesoNAM forecasts from ACTA, Inc. The 
current operational version of the MesoNAM is the 
12-km Weather Research and Forecasting (WRF) 
model. Based on the seasonal stratifications 
requested by the 45 WS and model availability, 
the MesoNAM forecasts were evaluated beginning 
with the October 2006 data, the first cool season 
month in the data set. The POR included three 
cool seasons. 2006-2007, 2007-2008 and 2008- 
2009; and two warm seasons: 2007 and 2008. 

The MesoNAM forecast files were provided to 
the AMU as space-delimited text files. Each file 
was based on a single model initialization time and 
was valid at a single point extracted from the 
model forecasts. This point was identified as 
"KXMR" in each file and represents the location of 
the CCAFS weather station, which is located near 
the center of CCAFS and is identified in Figure 1 
by the magenta square. The closest model grid 
point, which represents the point data used by the 
45 WS, is located 5.8 km southwest of KXMR over 
the Banana River. It is shown in Figure 1 as a 
green circle with a black dot and labeled "NAM". 


An inventory of the MesoNAM files in the POR 
revealed 128 missing files, or model runs, out of a 
possible 3772 files for the 943 days. Some days 
were missing less than four model runs while 
others were missing all four model runs. This 
resulted in a total of 910 days containing at least 
one model run. 

5. FILE FORMATTING 

Microsoft Visual Basic scripts were written to 
import the MesoNAM files into Microsoft Excel 
spreadsheets and reformatted to match the wind 
tower observation spreadsheets. This included 
converting the temperature and dew point from 
Celsius to Fahrenheit and moving rows and 
columns in the MesoNAM spreadsheets to match 
the wind tower spreadsheets. Visual Basic scripts 
were written to create an Excel workbook for each 
of the 910 days with at least one model run. Each 
workbook included up to four worksheets, one for 
each available model run, containing combined 
wind tower observations and MesoNAM data for 
each sensor on every tower. This resulted in a 
total of 24,570 workbooks. 
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6. VERIFICATION STATISTICS 


6. 1 . Temperature and Dew Point Example 


Verification statistics were calculated once the 
files were properly formatted and stratified. First 
the model bias was calculated for each model 
forecast against every observation. The means 
and standard deviations of the model bias for all 
stratifications (e.g., one month) as well as the root 
mean square error (RMSE) were calculated using 
the following equations: 


Where: 


1 

Bids Mean ~ ~ 


n 


1=1 


n = number of available model forecasts in 
any given stratification, 

/ = MesoNAM forecast of T, T d , wind speed 
or wind direction, and 

o = observed T, T d , wind speed or wind 
direction from each tower/sensor height. 


STDEV Bias = 




Where: 

n = number of available model forecasts in 
any given stratification, 

x = model bias of each forecast, 


Figure 2 shows a graph of the model bias of T 
and T d from a 1200 UTC model initialization at 
Tower 0020 at a sensor height of 6 ft for the month 
of January in the POR. Preliminary results indicate 
a periodic fluctuation was present in the model 
bias of T as can be seen in Figure 2. This result is 
consistent among the first three towers and all 
sensor levels analyzed as of the writing of this 
paper. The periodic fluctuation was observed in all 
four model runs per day. In January, the model 
had a positive T bias and a negative T d bias. The 
biases also decreased with forecast hour, with T 
coming closer to 0, but T d becoming more 
negative. 


Model T and T„ Bias 

12 UTC Initialization, lower 0020, 6 ft, January 



Forecast Valid Time from Model Initialization 


Figure 2. Graph of T and T d model bias from a 
1200 UTC model initialization at Tower 0020 and 
sensor height of 6 ft for January. The blue line is T 
and the red dashed line is T d . 


x = mean bias of each forecast period in any 
given stratification. 


RMSE = yjMSE 

n 

M5£ = ^(/i- 0i ) 2 

i=l 

Where: 

n = number of available model forecasts in 
any given stratification, 

/ = MesoNAM forecast of T, T d , wind speed 
or wind direction, and 

o = observed T, T d , wind speed or wind 
direction from each tower/sensor height. 


Figure 3 shows the model bias of T and T d from a 
1200 UTC model initialization at Tower 0020 at a 
sensor height of 6 ft for the month of May in the 
POR. The T bias was more negative than January 
and displayed a similar periodic fluctuation. The 
model T d bias was also negative in May. 


Model T end T d Bias 

12 UTC Initialization, Tower 0020, 6 ft. May 



Figure 3. As in Figure 2 but for May. 
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The model standard deviation and RMSE of T 
and T d are shown in Figure 4 and 5, respectively, 
for a 1200 UTC model initialization at Tower 0020 
at a sensor height of 6 ft for the month of January. 
Both graphs indicate the model performance 
degraded with time through the 84-hr forecast. 
This preliminary result of model degradation with 
time was also consistent among the three towers 
evaluated against the model thus far. 
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Figure 4. As in Figure 2 but for standard deviation of 
bias. 
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Figure 5. As in Figure 2 but for RMSE. 

6.2. Wind Speed and Direction Example 


Figures 6, 7 and 8 show the graphs of wind 
speed bias, standard deviation of bias and RMSE, 
respectively from a 1200 UTC model initialization 
at Tower 0020 at a sensor height of 54 ft for the 
month of January in the POR. These results 
indicate the MesoNAM forecasted wind speed 4-5 
kt too high throughout the entire 84 hour model 
forecast period for January. Preliminary results 
indicate the MesoNAM also forecasted wind speed 
too high at the other three towers evaluated thus 
far. Unlike T and Td, the wind speed forecast error 
did not increase significantly throughout the 84-hr 
forecast period but remained fairly constant. 


Modal Wind Speed Bias 

12 UTC Initialization, Tower 0020, 54 ft, January 



Forecast Valid Time from Model Initialization 


Figure 6. Graph of model wind speed bias from a 
1200 UTC model initialization at Tower 0020 and 
sensor height of 54 ft for January. 
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Figure 7. Same as in Figure 6 but for standard 
deviation. 



Figures 9, 10 and 11 show the wind direction 
graphs of bias, standard deviation of bias and 
RMSE, respectively from a 1200 UTC model 
initialization at Tower 0020 at a sensor height of 
54 ft for the month of January in the POR. The 
MesoNAM bias of wind direction was more 
negative, or to the right, of the observed winds. 
The standard deviation shows the bias was highly 
variable between 30 and 60°. 
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Model Wind Direction Bias 
12 UTC Initialization, lower 0020, 54 ft, January 
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Figure 9. Graph of model wind direction bias from a 
1200 UTC model initialization at Tower 0020 and 
sensor height of 54 ft for January. 

There was also a slight degradation in the wind 
direction forecast through the 84-hrforecast period 
for this tower as shown in the standard deviation 
of the bias and RMSE in Figures 10 and 11. 



of bias. 

7. FUTURE WORK 

In addition to the statistics presented in this 
paper, the 45 WS has requested conducting 
hypothesis tests for bias = 0, RMSE = 0 and if the 
composited bias and RMSE = 0. Also, if justified 
by the sample size, the data will be stratified by 


45°, 90° and 180° with the sector directions 
oriented to maximize discrimination between 
onshore and offshore flow. 


Model Wind Direction RMSE 
12 UTC Initialization, lower 0020, 54 ft, January 
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Figure 11. As in Figure 9 but for RMSE. 

In order to present the data to the LWO's in a 
manageable and user-friendly manner, a graphical 
user interface may be developed. A total of 9,240 
graphs would be generated based on the statistics 
being calculated for the monthly and seasonal 
stratifications for all sensors on all towers. The 
number of graphs could exceed 36,000 if the 
sample size is significant enough to stratify the 
data by sector. 
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